Mesh Interconnect Architecture - Intel

Intel’s mesh interconnect architecture is a multi-core system interconnect architecture that implements a synchronous, high-bandwidth, and scalable 2-dimensional array of half rings. Their mesh architecture has replaced the ring interconnect architecture in the server and HPC markets.

History

Throughout the 2010s as the number of cores on Intel’s high-end models continue to increase, the ring reached fairly problematic scaling issues, particularly in the area of bandwidth and latency.

A mesh networking topology in order to reduce the latency between nodes and increase the bandwidth.

For example, the 80 many-core Polaris processor from 2007 featured a mesh interconnect architecture with a 5-port router on each tile. Polaris was the very first research chip in the area of teraFLOP computing which eventually resulted in the Xeon Phi family of commercial processors.

In June 2016, Intel launched new Xeon Phi MIC microprocessors based on Knights Landing which was Intel’s first commercialized microarchitecture to implement the new interconnect architecture.

In mid-2017 Intel launched the Skylake server microarchitecture which featured also featured the mesh interconnect. This microarchitecture is found in their server (Xeon Scalable) microprocessors and the Core i7 and Core i9 HEDT parts.

Overview

Intel’s mesh interconnect architecture consists of a number of related concepts:

  • Mesh - the fabric, a 2-dimensional array of half rings forming a system-wide interconnect grid
  • Tile - a modular IP block that can be replicated multiple times across a large grid
    • Core Tile - a specific kind of tile that incorporates an Intel’s x86 core
    • IMC Tile - a specific kind of tile that incorporates an integrated memory controller
  • Caching/Home Agent (CHA) - a unit found inside the core tiles that maintains the cache coherency between tiles. The CHA also interfaces with the CMS
  • Converged/Common Mesh Stop (CMS) - A mesh stop station, facilitating the interface between a tile and the fabric

intel mesh overview.svg

Tiles are replicated in the X and Y axis as many times as desired. The type of tile depends on the design goals and target market. In theory any type of IP block can serve as a tile provided it’s modified to interface with the CMS. Each tile is associated with its own CMS which allows the tile to interface with the mesh. Every mesh stop at each tile is directly connected to its immediate four neighbors – north, south, east, and west.

intel mesh cms links.svg

he mesh itself consists of a 2-dimensional array of half-rings. Every vertical column of CMSs form a bi-directional half ring. Similarly, every horizontal row forms a bi-directional half ring.

Horizontal bi-directional half rings
intel mesh cms links (horizontal).svg

Vertical bi-directional half rings
intel mesh cms links (vertical).svg

Operations

A packet follows a simple routing algorithm:

  • Packets are 1st routed vertically
  • Packets are then routed horizontally

A packet originates at a tile (e.g. from the CHA) or some an I/O peripheral. It enters the fabric at its local Mesh Stop (CMS). The packet is then routed along the vertical half ring, either north or south, always taking the shortest path. Once the packet reaches its destination row, it will be taken off the vertical half ring and placed on the horizontal half ring where it will continue to the destination tile. Once the packet reaches the destination tile, it will interface back with the tile via Mesh Stop.

Example

Consider the 5 by 6 example mesh below. For a packet to go from the tile labeled ‘Start’ to the IMC, the packet will leave the core via the CHA and onto the mesh via the CMS. The packet will then be routed through three stops going north. The packet will then be taken off the vertical half ring and placed on the horizontal half ring where it will continue three additional stops going east to the IMC.

intel mesh example start.svg

It’s important to note that the return path may not be the same route as before. For example, as in this situation, a response from the IMC will be routed via the vertical half ring first to the very south-east corner tile. The packet will then be taken off the vertical half ring and placed on the horizontal half ring where it will go west until it reaches the destination tile.

intel mesh example return.svg